Go/No Go Meeting 2022
From human to data-driven decision-making …
… where black boxes are recipe for disaster.
Machine Learning algorithms and automated decision-making systems are increasingly being deployed in Finance and Economics. While the introduction of artificial intelligence (AI) promises to generate many beneftis, it also entails great risks and challenges for regulators, market participants and society at large. In this PhD project we plan to identify and tackle some of these challenges through methodological contributions and applied work. We focus in particular on Probabilistic Models and Counterfactual Reasoning.
Ground Truthing
Probabilistic Models
Counterfactual Reasoning
CounterfactualExplanations.jl is a package for generating Counterfactual Explanations (CE) and Algorithmic Recourse (AR) for black-box algorithms. Both CE and AR are related tools for explainable artificial intelligence (XAI). While the package is written purely in Julia, it can be used to explain machine learning algorithms developed and trained in other popular programming languages like Python and R. See below for short introduction and other resources or dive straight into the docs.
LaplaceRedux.jl (formerly BayesLaplace.jl) is a small package that can be used for effortless Bayesian Deep Learning and Logistic Regression trough Laplace Approximation. It is inspired by this Python library and its companion paper.
TLDR: We find that standard implementation of various SOTA approaches to AR can induce substantial domain and model shifts. We argue that these dynamics indicate that individual recourse generates hidden external costs and provide mitigation strategies.
Description: In this work we investigate what happens if Algorithmic Recourse is actually implemented by a large number of individuals. The chart below illustrates what we mean by Endogenous Macrodynamics in Algorithmic Recourse: (a) we have a simple linear classifier trained for binary classification where samples from the negative class (y=0) are marked in blue and samples of the positive class (y=1) are marked in orange; (b) the implementation of AR for a random subset of individuals leads to a noticable domain shift; (c) as the classifier is retrained we observe a corresponding model shift (Upadhyay, Joshi, and Lakkaraju 2021); (d) as this process is repeated, the decision boundary moves away from the target class.
We argue that these shifts should be considered as an expected external cost of individual recourse and call for a paradigm shift from individual to collective recourse in these types of situations.
The results in Figure 2 look great!
But things can also go wrong …
The VAE used to generate the counterfactual in Figure 3 is not expressive enough.
The counterfactual in Figure 4 is also valid … what to do?
TLDR: Using Linearized Laplace Redux we can compute predictive uncertainty estimates for any neural network (Daxberger et al. 2021). By minimizing predictive uncertainty we can create realistic counterfactual explanations (Schut et al. 2021) without the need for surrogate generative models.
Description: We propose LaplaCE: an effortless way to produce realistic Counterfactual Explanations for deep neural networks (DNN) using Laplace Approximation (LA). To address the need for realistic counterfactuals, existing work has primarily relied on separate generative models to learn the data generating process (e.g. Joshi et al. (2019)). While this an effective way to produce plausible and model-agnostic counterfactual explanations, it not only introduces an significant engineering overhead, but also reallocates the task of creating realistic model explanations from the model itself to the surrogate generative model. Recent work has shown that there is no need for any of this when working with probabilistic models that explicitly quantify their own uncertainty. Unfortunately, most models used in practice still do not fulfill that basic requirement, in which case we would like to have a way to quantify predictive uncertainty in a post-hoc fashion. Recent work on Bayesian Deep Learning has shown that LA can be used effectively in this context. By leveraging this finding we show that it is possible to generate realistic counterfactual explanations, without the need to restrict the class of models or rely on a separate generative model.
Potential venues1: FAccT (Dec ’22), AIES (March ’23), NeurIPS (May ’23), SaTML (Sep ’23)
TLDR: Using a gamified experiment we test if SOTA XAI methods can actually help users to understand the workings of a black-box model.
Description: Do Counterfactual Explanations actually help users to understand the workings of a black-box model? In this work we investigate this question through gamified experiments. The idea is to set up an experiment as follows:
If the XAI method is useful, the discrepancy between user guesses and neural network prediction should diminish over time. This project idea is inspired by an AIES 2022 paper that employs a similar framework (Dai et al. 2022).
Potential venues: AIES (March ’23), JuliaCon (April ’23)
TLDR: The automated decision-making system used by the Dutch tax authorities is opaque not because of its complexity, but rather by design. The goal of this work is to explore ways to explain such black-boxes through counterfactuals.
Description: The Dutch childcare benefits scandal involved involved false fraud allegations based on an automated decision-making system (ADMS) used by the tax authorities. The ADMS was essentially a collection of spreadsheets containing hard-coded rules. The sheer quantity of spreadsheets has made it difficult for experts (Cynthia) to disentangle the inners workings and hence understand the behaviour of the ADMS. We can think of this a non-conventional black-box that is opaque not because of its complexity, but rather by design. We believe that these types of ADMS are still widely prevalent in industry and therefore should be considered as a different kind of threat to AI integrity. The goal of this work is to explore ways to explain such black-boxes through counterfactuals. This is a challenging and ambitious task, but a few strategies come to mind: 1) use brute force to search counterfactuals; 2) use a Growing Spheres (Laugel et al. 2017) to generate counterfactuals; 3) derive a decision tree from the spreadsheets and generate counterfactuals for the tree.
Potential venues: FAccT 2024
Description: This fall I will give a seminar about Counterfactual Explanations and Algorithmic Recourse at the Bank of England. Bank researchers are interested in applying CE and AR to their bank risk prediction models.
Potential venues:
Description: The literature on Counterfactual Explanations almost exclusively focuses on classification problems. In Finance and Economics, however, the overwhelming majority of problems involve regression. Hence it is perhaps not altogether surprising that practitioners and researchers in these fields are largely unfamiliar with the CE and instead typically rely on surrogate explanations like LIME and SHAP to explain black-box models. Using Spooner et al. (2021) as a potential starting point, I would be interested in exploring how state-of-the-art CE approaches can be applied to regression problems.
Potential venues: -
Counterfactual Reasoning and Probabilistic Methods for Trustworthy AI with Applications in Finance – Patrick Altmeyer